International Review of Research in Open and Distributed Learning 
Volume 17, Number 1 


January-2016 


Challenges of Transitioning to an e- 
learning System with Learning Objects 
Capabilities 



Miroslava Raspopovic, Svetlana Cvetanovic, Aleksandar Jankulovic 

Belgrade Metropolitan University, Serbia 


Abstract 

Higher education institutions that have implemented blended and/or online learning need to be 
on the forefront of the cutting edge technological and educational advances in order to remain 
competitive and innovative. This task is usually very difficult due to the budget constraints that 
many institutions have. Keeping up with the current technological advances often brings not only 
financial challenges, but also transitional challenges that may put at risk learning quality and 
reputation of the institution, as well as performance of students. This work describes the features 
of the system, and the results and challenges of transitioning to an e-learning system that displays 
learning materials through a sequence of reusable learning objects (LOs) from the system that 
does not have these capabilities. Based on the research of existing methodologies and the results 
of this research, this work proposes a methodology for transferring into an LO-oriented e- 
learning system. Findings of this work reveal advantages, disadvantages, and potential obstacles 
of e-learning system implementation with LOs and give an overview of suggestions for 
implementation improvements. These suggestions are given based on evaluation of the 
implementation of an LO-oriented e-learning system, after the transition from the traditional 
non-LO oriented e-learning system. 
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Introduction 

Rapid developments of information and communication technologies (ICT) are introducing new 
opportunities in the development of e-learning systems. Traditional e-learning systems are used 
to organize and publish learning materials, typically in the format of written lectures or video 
tutorials. However, the tendency for the new systems is leaning towards the personalization and 
interactivity of learning materials for each student (Chun, 2004). One of the ways to achieve this 
is by segmenting learning materials into smaller units, with the goal of being able to assemble 
lessons based on the already existing materials. In the context of e-learning materials, these 
smaller and reusable units are referred to as learning objects (LO). LOs can be used to guide 
students' learning by determining the sequence of LOs for each user based on their study 
curriculum and prerequisite knowledge (Cvetanovic & Raspopovic, 2012). LOs represent a good 
candidate in online learning for enhancement of learners' effectiveness, performance and 
experience. Usage of LOs and their repositories are said to offer many benefits to learners 
(AlMegren et al., 2013). 

Much work has been done on analyzing how different types of e-learning systems influence 
learning experience and effectiveness. The focus of this paper is to identify potential problems 
and challenges in implementation of a new e-learning system with LOs when transitioning from a 
traditional e-learning system. The goal of this paper is to discover advantages, disadvantages, and 
potential obstacles of this transition and to propose suggestions for improvements. Transitional 
challenges that one institution may face are of interest, as this situation may pose difficulties that 
are not encountered when introducing the first e-learning system in the academic institution. It is 
considered important to integrate technology solutions in the process of teaching and learning 
and that the whole process should be appropriately designed (Cukusic et al., 2010). 

According to literature research, much work has been done to propose different methods for 
implementing new information systems. While using a traditional Waterfall method can be time 
consuming and can put a big burden on the budget, new methods have been proposed to make 
this process more efficient. In recent decades, “agile” software development methodologies have 
been put forth as an alternative to traditional “waterfall” implementation; an “agile” 
implementation methodology has been put forth (Chan et al., 2009, Brhel et al., 2015). 

Agile implementation focuses on short cycles (sprints), in order to make faster progress, and to 
improve monitoring the progress of the implementation. The idea is to complete one part of the 
system at the end of each system, and to allow re-evaluation of the project’s direction. This agile 
implementation methodology is usually used by SAP in the implementation of their Enterprise 
Resource Planning (ERP) packet systems. According to the SAP proposal (SAP AG, 2011), the 
implementation should start by identifying the most important functionalities (called “must have” 
functions) without which new systems cannot be put into operation. The next iteration allows for 
delivering the “nice to have” functions which are defined by the so called “delta list”. We adopted 
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this agile approach and improved this agile process to be more suitable for the implementation of 
the e-learning systems. 

The paper is organized as follows. Section 2 presents e-learning systems and their integration 
with learning objects, along with methods of evaluation of e-learning systems. Section 3 presents 
the research methodology used for result analysis, while Section 4 presents the results and 
discussion of this research. Section 5 concludes the paper. 


E-learning systems and their integration with LOs 

Progress in the field of e-learning has been very slow due to the problems that are mostly 
associated with the poor interface design of e-learning systems (Zaharias, 2005). On the other 
hand, research confirms the high importance of design and employment of methodologies, tools, 
and learning environments for stakeholders (both teachers and students), as well as usability 
evaluation of e-learning applications (Granic, 2008). A significant and important part of an e- 
learning system is the learning management system (LMS). LMS is the “front face” of the entire 
system and what users are able to see and interact with. An LMS is designed to organize and 
regulate the administrative tasks of schools and other educational organizations (Esther, 2008). 
According to Watson and Watson (2007), an LMS provides functionalities beyond instructional 
content such as management tracking, personalized instruction, and integration across the 
system. However, the LMS does not constitute the entire e-learning platform, especially when 
considering e-learning systems that use LOs. In addition to an LMS, an e-learning platform that 
can work with LOs has a system for creation and editing of LO content; it allows retrieval and 
reuse of LO, and delivery of appropriate LO to the LMS. 

Not every LMS supports implementation and presentation of learning materials through a series 
of LOs. The problem with LO-oriented systems is a potential gap between designing different 
levels of learning and designing and planning a higher-level lesson or course. There are many 
examples that reflect the problems, from course design down to LOs design. The LDSE project 
provides the support for learning design including specific learning activities, lessons, and full 
courses. LDSE points out that designs at the ‘higher’ layer should incorporate ‘lower layer’ designs 
(LDSE, 2009). JISC D4L program (Beetham, 2008) set out the model to address linking between 
higher-level designs, which include course/lesson planning and LOs design. This model was 
represented as four layers: course design at the highest level, session or lessons planning, activity 
design (things learners actually do), and designing LOs. The key weakness of this model was the 
separation between layers of learning activity and learning objects that was treated as 
unsustainable. 

In designing a learning object layer, two categories of design principles can be implemented: 
structural and pedagogical. The aim of structural principles is to create self-contained LOs with 
reusable content. Structural design principles include cohesion and de-coupling (Boyle, 2003). 
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On the other hand, the aim of the pedagogical design principle is to create rich interactive 
learning experiences (Boyle, 2008). One system that graphically supports the design approach at 
the lesson/course level, and is inspired by IMS LD specification (IMS LD, 2003), is LAMS system 
(Dalziel, 2003). LAMS is based on the selection and sequencing of different pedagogical 
functions for presentation, discussion, and previously designed LOs. 

For effective implementation of LOs within an e-learning system, it is recommended to have a set 
of assigned metadata for each LO. Metadata enable LO to be described, indexed, and searched. 
Several metadata standards have emerged and can be used to describe LOs and their resources: 
IEEE LOM, ARIADNE and IMS. However, the high quality design and development of LOs, 
enabled by pedagogical principles, is more important than the LO's metadata description. A new 
concept of Generative Learning Objects (GLO) was introduced with pedagogical principles in LO 
design (Boyle, 2006a; Boyle, 2008). GLO establishes an underlying learning design that can be 
created, inspected, and adapted to meet learners' needs. Learning design tools can be classified in 
four groups: authoring, ontology-based, standards-based, and generic form-based tools 
(Charlton, 2010). 

There are many authoring tools that allow for the creation and reusability of learning design. 
Authoring tools can incorporate just structural, or a combination of both structural and 
pedagogical design principles. There are some authoring tools that should be mentioned such as 
Articulate Presenter, Snap, Raptivity, Xerte, Course Lab, Atutor, GLO Maker, and Microsoft 
content developer (LCDS). Generally, authoring tools should fulfill all features including 
pedagogy, reusability, and technology (Diwakar, 2013; Dag, 2014). For the purpose of this study, 
the DITA authoring tool was used. 

Ontology enables the organization of learning materials around small pieces of semantically 
enriched LOs that can be easily organized into customized learning courses and delivered on 
demand to the user (Nejdl, 2001). It allows for associations between multiple LO repositories and 
solves the problem of metadata misconception. Ontologies are classified in three categories: (1) a 
content-domain ontology that solves the problem when semantically identical concepts are 
expressed by different keywords, (2) a content structure ontology that explicitly defines LO 
structures that may be useful in many cases when there is a need to reuse a specific part of an LO, 
rather than using the LO as a whole, and (3) context (didactic) ontologies formally specifying the 
educational/pedagogical role of a content unit (Stojanovic et al., 2001). 

For the purpose of this study, IBM’s Darwin Information Typing Architecture (DITA) was 
implemented as a content structure ontology. DITA provides two structure components: topics 
and maps. A topic can be used to describe an overview, summary, or particular concepts or tasks 
as parts of a lecture, exercises, assignments, case studies, etc. The map is used to specify the 
chosen learning design model. In recent years, learning design has moved away from a sequential 
waterfall model, where output from one phase serves as an input to the next, and becomes a more 
fluid and nonlinear process that relies on frequent evaluation and alternation. In this view, 
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learning design should be seen as a complex adaptive system where one component in the design 
process influences all other components, not simply a sequential element (Ritchie, 1999). 

E-learning evaluation 

E-learning evaluation is not a one-time activity but rather a complex process parallel to e-learning 
development as well as its implementation (Dvorackova & Kostolanyova, 2012). A large number 
of e-learning evaluation models exist. One of the most cited and implemented is the DeLone and 
McLean model (Raspopovic et al., 2014). The evaluation of an LMS must consider its usability 
and accessibility, as well as its didactic effectiveness in order to achieve educational goals. If an e- 
learning system is not usable enough, it obstructs students' learning and the learners would spend 
more time learning how to use the software rather than learning the contents (Wong, 2003). The 
usability of the system is of great importance for e-learning implementation, and therefore, it is 
also important for the transition from one system to another. It is imperative to research and 
implement an appropriate methodology for the system usability. 

Several studies have investigated the usage of learning objects and their impact on improving the 
teaching and learning process (Boyle, 2003; Cochrane, 2005; Van Zele, 2003). Khor (2014) 
showed that users’ perception has a significant effect on the acceptance and adoption of usage of 
learning objects. 

According to literature research, there are a number of methodologies for the evaluation of e- 
learning systems: the methodology for systematic usability evaluation - SUE (Ardito et al., 2006); 
the hierarchy structure for evaluating web-based e-learning systems - WELS (Shee & Wang, 
2008); the hexagonal e-learning assessment model - HELAM (Ozkan & Koseler, 2009); the three 
stages model (Cukusic et al., 2010); and a model for using e-learning systems for sharing 
experiences, communicating with others, and getting feedback from student’s learning 
experiences (Wang & Chiu, 2011). 

The reference model is the Layer 3 of the Institute of Electrical and Electronics Engineers (IEEE) 
Learning Technology Standards Committee (LTSC) - standard IEEE P1484.1 LTSA defines three 
components: Processes (boundaries, services, inputs and outputs of the learning system), Stores 
(repository of data that can be accessed by users), and Flows (connectivity and the type of 
information exchanged). The standard IEEE P1484.1 LTSA as a mode defined by an authoritative 
institution is proposed as a framework that encourages portability and reusability, and provides a 
good basis not only for architectural and functional design, but also for evaluation of candidate 
software products for a Virtual University Information System and e-learning platform (O’Droma 
et al., 2003). 

International ISO standard ISO/IEC 19796-1, developed in consensus by the International 
Standard Organization (ISO), defines standards in learning, education, and training in general. 
The ISO/IEC 19796-1 standard was developed by the Working Group 5 “Quality Assurance and 
Descriptive Frameworks” of the standardization committee ISO/IEC JTCi SC36. The quality 
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standard contains the reference process model “Reference Framework for the Description of 
Quality Approaches” (RFDQ) to help stakeholders in learning, education, training, and especially 
in e-Learning or blended learning to document and (re-)define their everyday business and 
processes (ISO/IEC t9796-r:2005; Stracke & Hildebrandt, 2007). The Reference Process Model 
from the ISO quality standard RFDQ (ISO/IEC 19796-1) consists of seven process categories and 
38 processes: 

• Needs analysis (Initiation, Stakeholder Identification, Definition of objectives, 
Demand analysis); 

• Framework analysis (Analysis of the external context, Analysis of staff 
resources, Analysis of target groups, Analysis of the institutional and 
organizational context, Time and budget planning, Environment analysis); 

• Conception/design (Learning objectives, Concept for contents, Didactical 
concept/methods, Roles and activities, Organizational concept, Technical 
concept, Concept for media and interaction design, Media concept, 
Communication concept, Concept for tests and evaluation, Concept for 
maintenance); 

• Development/production (Content realization, Design realization, Media 
realization, Technical realization, Maintenance); 

• Implementation (Testing of learning resources, Adaptation of learning 
resources, Activation of learning resources, Organization of use, Technical 
infrastructure); 

• Learning process (Administration, Activities, Review of competency levels); 

• Evaluation/optimization (Planning, Realization, Analysis, Optimization/ 
Improvement). 

In our research we analyze our suggested methodology for implementation of LO in e-learning 
systems by three components defined in the IEEE standard and through the seven process 
categories of EN ISO/IEC 19796-1. Analysis is conducted through a students’ survey. 

Functionalities of e-learning systems with and without LOs 

An e-learning system that does not have the capability to present learning materials through LOs 
(Type A systems) is designed as a system with the static and non-LO oriented content. Learning 
materials for Type A systems contain the materials that are presented and prepared in the same 
manner for all students. The objective of such system is usually to transfer the knowledge, 
information or skills from the teacher to students. 
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LO-oriented e-learning systems (Type B systems) are based on finely granular LOs. LOs represent 
modular and logically independent reusable units, even though they may be related to other LOs 
in a larger scope. As discussed by Sicilia & Garcia (2003), reusability is an essential and arguably 
the most important characteristic of the reusable LOs. Drucker (2000) compares the 
characteristics of non-LO oriented systems and LO systems through several dimensions: delivery, 
responsiveness, access, symmetry, authority, personalization, adaptability, etc. By analyzing these 
two types of systems, the author concludes that they show completely different characteristics. 

According to the research of Baki and Cakirogly (2010) within learning objects in high school 
mathematics classrooms, the students’ views and approaches about the used objects have 
indicated that the innovations in courses by using LOs had a positive influence. Students have 
made positive comments about the motivational and learning themes, enhanced interactivity, and 
higher visual potentials. The negative views were generally about technological conditions and 
time-consuming, similar examples. 

In the study of McCormick & Li (2005), teachers considered that LOs were beneficial tools for 
learning and that they could be used for scaffolding to understand concepts. Teachers were 
interested in using LOs in their classrooms. The preferences about teachers' LO use are consistent 
with the results of case study research about LO use in classroom environments (Ilomaki, 
Lakkala, & Paavola, 2006). The preferences are shaped through student, design, and content- 
based causes. The results about preferences indicate that organizing, structuring, and guiding 
processes of the course are crucial. In conclusion, as Kay & Knaack (2007) pointed out, students 
may benefit from LOs if they are comfortable with LOs offering good learning control, useful 
content, and clear instructions. 

In order to describe differences between Type A and Type B systems, two categories of 
functionalities were considered: preparation and usage of learning materials. Functionalities and 
differences between these two types of systems are described in Table 1 and are mainly based on 
the literature review and previous experiences. Even though they support the same categories of 
functionalities, the teaching and learning scenarios in Type A and Type B systems are performed 
in different manners. 

Table 1 

Comparison of functionalities between Type A and Type B systems 
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Functionalities 

Type A 

Type B 

Preparation of learning materials 

Creation of the 

learning 

materials 

Creation, organization and 
control of learning material are 
the responsibility of teachers. 
Teacher and reviewers evaluate 
learning material. Students have 
some influence in the published 
content, and this influence is 
usually reflected after the course 
ends, in the process of 
preparation for the next session. 

In Type B systems, initial development 
of course materials consists of several 
activities that relate to: (a) preparation 
of high quality LO by applying 
pedagogical design principals and 
appropriate LO methodology, (b) 
aligning the LO structure to the 
appropriate ontology structure and 
determining metadata for LO - teacher 
"enriches" each LO with the metadata, 

(c) lesson design as to which role is to 
orchestrate previously created LOs with 
different learning activities, (d) course 
design by applying the same principles 
used for lesson design. 

Format of 

learning 

materials 

Course material consists of 
lectures in the written format, 
which is typically in the book 
chapter format, presentations 
with voice and/or video 
recordings in the lecture style, 
course assignments, assessments, 
and discussion forums. 

Instead of lectures written in the book 
chapter format, in Type B system there 
are lectures consisting of different 
combination of LOs. In order to be able 
to support different learners’ needs and 
interests, a repository of sequenceable 
multimedia LOs is necessary. The 
sequencing of LOs allows for more 
effective organization, presentation, 
and search of LOs. 

Learning design 

Type A systems support only 
sequential instructional design. 
From the student’s perspective, 
the student can choose one of 
only two learning styles (visual or 
auditory). Visual style enables 
student to read enriched text 
with graphics, illustrations, 
images, tables, etc. Auditory style 
allows students to listen to 
lectures. It is typical to have a 
heterogeneous group of students 
in one classroom with different 
levels of background knowledge; 
however, all students have to 
study the same learning contents. 

Type B system is suitable for an 
adaptive instructional design model 
that relies on frequent evaluation and 
alternation. Students can access LOs in 
whatever sequence, not only in a linear 
one. This means that the system 
exposes more possibility of direct 
access to the chunk of knowledge 
contained in LOs. 

Learning materials usage 

Learning 

activities 

All assignments, assessments, 
and lectures are published as 
separate activities. Due to this, 
the learning material is not easily 
adaptive. 

Due to the possibility of combining 
different LOs in different manners at 
the lesson level, Type B system is able 
to provide more flexibility to 
instructors when dealing with different 
types of students. 
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Interaction with 
instructor 

Training and learning occur as 
separate activities, with defined 
starts and stops, so that students 
are not able to keep in touch with 
the instructor during the learning 
process. In addition, students’ 
interactions with the instructor is 
asynchronous, as the student is 
not able to collaborate or interact 
with the instructor directly. 

Student is able to exchange 
opinions with the instructor only 
by emails, conference messages, 
and forums. 

Learning is the integral component of 
the training process, and it is 
performed through the iterative 
process. During the iterations, students 
are able to collaborate directly with the 
instructor and other students in order 
to provide feedback that enables the 
instructor to continuously change and 
adapt learning content. 

Learning 

assessment 

Assessment is done through 
grading of assignments and tests. 
These assessments usually do not 
influence adjustment of learning 
materials. 

During the learning process, instructor 
can analyze the current knowledge level 
of a student based on results of 
students’ knowledge self-assessments 
and adjust learning material to their 
needs. 


Methodology for transitioning to a new e-learning 

system 

When transitioning from one e-learning system to another, one should consider an appropriate 
transition model that will allow learners who are used to using a Type A system to maintain 
and/or increase effective learning. In our case, the goal was to transition from the system that 
views learning materials as a whole, to a system that allows sequencing of different learning 
objects (LO), along with different learning activities. As previously mentioned, in this paper the 
system that can work with LOs is referred to as an LO-oriented e-learning system (Type B), and 
those that cannot as non-LO oriented (Type A). 

Implementation of the new system and the transition from Type A to Type B system can be done 
in different ways. Since implementing a new e-learning system using the Waterfall method can be 
time consuming and costly, in our approach we combined Waterfall and agile methods resulting 
into a hybrid model. The goal is to allow for the implementation of the new system to be 
structured, adaptable, and flexible. 

Hybrid transitional model 

Based on the Waterfall and agile methods, we suggested a Hybrid transitional model, as shown in 
Figure l. In this model the first phase of the Waterfall method is kept, as it is necessary to plan 
organizational and system changes, which have to be in line with the institution’s goals and 
mission. In this model, we assume that learners' goals are set and are in alignment with 
institution’s goals. In other words, we assume that the course is already planned out, as it existed 
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in the old system. If the course is new, the additional step of planning the course outline and 
learning objectives should be taken into consideration. Once the organizational and system 
changes are prepared, the backlog of “must have functions” are determined. These are the 
functions with highest priority, without which the new system cannot be put into operation and 
which are primarily related to publishing learning material in a Type B system based on learning 
material in the Type A system. 


PreDare changes 

Organizational changes 

- Agree on goals 

- Determine and agree on necessary organizational 
changes 

System changes 

- Identification of all operation processes 

- Plan out all the "must-have" functionalities of the 
system that will be used to achieve the set goals 

- Choose the system to be used (based on the set 

- functionalities) - 


— 


Functionalities 

backlog 


“must have” Setup base 

functionalities course 


Train users on the 
base course 


Convert learning 
content to 00 
oriented content 



Figure 1. Using a hybrid model (Agile+Waterfall) for transitioning to a new e-learning system 

“Must have” functions were determined based on the organization’s set goals, objectives, and 
needs. Choosing vital functionalities of the Type A system to be part of “must have” functions for 
a Type B system can be an option in order to allow students to transition and adjust easier with 
the least possible interruptions in the quality of their learning. As “must have” functions were 
mainly based on the previous experiences and literature review, it is the intent of this work to 
analyze whether these choices were good, and how they can be improved. 
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In the case of a Type B system, “must have” functions were chosen to allow for an adaptive model 
of learning design at lesson and course level, and an adaptive teaching/learning process 
performed through the iterative process. 

The following functions were determined as “must have functions” for the Type B system: 

• LO authoring - The aim of LO authoring is to create high-quality LOs based on 
both structural and pedagogical design. In order to create reusable LOs, 
cohesion and decoupling with agile methodology of LO development 
(Cvetanovic & Raspopovic, 2014) were used. Furthermore, to capture and 
organize LOs content, a DITA framework for structural writing was used. Since 
the LO structure and DITA topic structure are very similar, each LO was 
presented as a DITA topic that is defined as an aggregation of following 
components of an LO: title, prologue, content, related links, glossary, and 
learning sub-objects. Prologue contains information necessary for LO searching 
and retrieval such as categorization to which curriculum and academic program 
this LO was developed for, difficulty level of the learning content, and 
keywords. In order to provide prologue information, each LO is "enriched” with 
the metadata that describes what the learning content is about, according to a 
previously defined content ontology. Furthermore, LO content was organized 
into sections: each section has its own title and body which can contain 
paragraphs, images, figures, formulas, and links to external resources (PDFs, 
zips, multimedia files, programming source code, etc.). Keeping in mind that 
there are various authoring tools, for the purpose of this work, the QDITA 
authoring tool was used. QDITA is a customized authoring tool based on an 
open source DITA authoring tool (Guduric et al., 2013). 

• Learning (instructional design) - For higher level designs, two types of maps 
were used: lesson maps and course maps. Lesson maps, in addition to different 
topic types (specialized for overviews, content, summary), include two types of 
activities: students’ self-assessment such as tests containing open questions, 
true/false, single select or multiple answer type questions, and activities that 
enable communication and interaction, for example, chat, wiki, forum, etc. In 
the initial phase of the project, basic functions of the old system were 
maintained and included in “must have” functions, which related to interactive 
questions in test format for self-assessment and discussion forums. The course 
map's purpose is to organize a sequence of learning materials, including LOs, 
weekly assessments (test, homework, exercises), and course projects. For lesson 
and course preparation, the LAMS system was used. 

• Publishing - The publishing methods were chosen based on the previous 
learners’ habits. Output formats were mainly chosen to be published on web 
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using XHTML, as well as in PDF format. LAMS can be automatically 
instantiated as a Web-based learning session using the LAMS delivery 
infrastructure. 

Before learning material is developed for an individual course, it is recommended that the base 
course is developed. The base course will be used to train other instructors on how to adapt and 
enhance their course materials for the new system. 

In order for an e-learning system with LOs to be successful, it is necessary to use high-quality LOs 
that are structured, adaptable, and ultimately reusable in different contexts. Even though one 
may try to maintain modularity and logical independence of each LO, in actuality, group of LOs 
are usually developed in order to meet the needs of a certain learner, usually having in mind a 
particular course. These LOs are developed by course instructors and/or multimedia developers 
(Boyle et al., 2006). While course instructors are mainly in charge of developing content and 
pedagogical methods for presenting the materials, the multimedia developer's role is to enhance 
LOs with visual presentation under the instructor's supervision. What should be kept in mind is 
that interaction between course instructors and multimedia developers in this model represents 
an iterative process. Once the course instructor develops the content for LO, this content is 
forwarded to the multimedia developer. Another step can be incorporated in this iterative process 
in which the learning content should be first forwarded to the reviewer, who should be a domain 
expert. Even though involvement of the reviewer can ensure and improve academic quality of 
LO’s content, at the same time it may increase the cost of the entire process. Once the instructor 
forwards the LO, the multimedia developer develops the multimedia based on the given 
specifications. In this iterative process, the resulting LO is given back to the instructor for a 
review. This iterative process between instructor, reviewer, and multimedia developer continues 
until all specifications and criteria are met, and the LO is published (Cvetanovic & Raspopovic, 
2014). 

Once LOs are published, the instructor can receive feedback from learners about the published 
material. More importantly, quality assurance and evaluation can be completed through 
involvement of the learner in the entire process, so that learning materials are tested with a 
certain group of students. This is referred to as “use before reuse” (Boyle et al., 2006). 
Considering that these LOs are used in a “live” scenario, these evaluations can be done from a 
one-week period to the entire semester. This feedback and analysis should give a better input 
about things that should be changed, adapted, and enhanced throughout the semester. More 
importantly, from this type of evaluation, conclusions can be made as to how the entire system 
can be improved. In this evaluation process, “nice to have” requirements and functionalities are 
determined. These are then used for the next phase of system improvements. “Nice to have 
functions” were determined based on the faculty and student evaluation throughout the semester, 
and finally at the end of the semester, after the evaluation of the entire system was completed. 

Our proposed hybrid model elements are based on ISO dimensions (Table 2). 
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Table 2 

ISO dimensions and hybrid model 


ISO dimension 

Proposed hybrid model 

Needs analysis and Framework analysis 

Preparation of organizational and system changes 

Conception/design 

Setup base course and train users for the base course 

Development/production 

Development LO content, peer review of LO, 
multimedia development 

Implementation and Learning process 

Student learning using weekly published lectures 

Evaluation/optimization 

Assessment using performance metrics 


Research Methodology 

In this paper we propose a model for transitioning from a non-LO system (Type A) to an e- 
learning system that supports generation, sequencing, and publishing of LOs along with different 
activities (Type B). The impact on students’ learning and study process by transferring from a 
Type A to a Type B system was analyzed. The purpose of this study was to analyze effective ways 
of changing an e-learning system and how these changes influence satisfaction, learning 
experience, and usage of the system. Furthermore, it is of interest to analyze how this transition 
can contribute to student motivation and satisfaction. The examination of this evaluation was 
expected to reveal good practices, obstacles, and needs for future improvements. In the conducted 
case study at the academic institution, the Type A system was used for eight years, before 
deciding to change to a Type B e-learning system. During the first year of the transition to the 
Type B system, the process of transferring from the Type A to the Type B system was documented 
and an initial evaluation was performed. To illustrate the outcome of applying these practices, the 
illustrative case study examined the following: 

• Students’ degree of satisfaction with the new system 

• Student satisfaction with published materials 

• Students’ motivation and driving forces to be active (i.e., communication with 
peers, interaction with the instructor, etc.) 

Current literature and methods of developing, implementing, and evaluating information and e- 
learning systems were employed (Boyle et al., 2006). This resulted in adapting an agile 
methodology for the development of course content in form of LOs and overall implementation of 
the Type B system, presented in the following section. 
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Faculty was asked to develop their courses in the form of a sequence of LOs and interactive 
activities. Both online and traditional students were asked to evaluate these LOs by using these 
materials in their courses for the entire semester. Data was collected throughout the semester 
through student interviews and at the end of the semester through a web questionnaire. Data 
from student interviews was used to improve the quality of learning content throughout the 
semester. Statistics comparing students’ usage of the system and satisfaction with the new e- 
learning system was analyzed. 184 students participated in this evaluation, out of which 93 were 
traditional, and 91 online students. These results were compared to the previous year’s results, 
collected prior to the introduction of the Type B system. As a reference, the analyzed Type A 
system was an Oracle iLearn platform, and Type B was a Learning Activity Management System 
(LAMS) with DITA standard. It should be noted that even though illustrative examples are used 
by referencing certain e-learning systems and technologies used, it is not the intent of this work to 
focus on tools and technologies, but rather on the model for a systematic transition from a non- 
LO oriented system to an LO-oriented e-learning system. Based on the evaluation analysis, 
recommendations for improvements and future research are given. 


Analysis of success factors of the implemented 

transition 

The evaluation of the Type B system and its implementation was done continuously throughout 
the semester with traditional and online students. Traditional students used learning materials on 
the e-learning system in addition to traditional lectures and labs in the classroom in blended-style 
learning, while the online students used the same materials without attending the lectures. Each 
lesson on the e-learning system consisted of a sequence of different topics (LOs), interactive 
questions in test format, discussion forums, assessment tests, homework assignments, exercises, 
projects, and summarized learning materials in a written lecture form in PDF format. At the end 
of the semester, students were asked to give their opinion on their learning experience via a 
survey. Each survey question offered scaled marks from 1 to 5 (1 -unsatisfactory, 2- sufficient, 3- 
average, 4-very good, 5- excellent ). 184 students participated in this evaluation; 93 were 
traditional, 91 were online students, while 108 were men and 78 women. 

The student survey includes analysis of IEEE components Processes and Flows (survey questions 
for evaluation of students’ degree of satisfaction shown in Table 3) and Stores (survey questions 
for evaluation of degree of satisfaction about the quality of learning materials published on the 
Type B system shown in Table 6). 

Based on the average marks on the survey of students’ satisfaction (Table 3), students were most 
positive about interaction and additional consults with instructors, 3.96 ±1.08, and their current 
learning motivation, 3.72 ±1.18. Similarly, they expressed their positive opinion about weekly 
course assignments, which helped them during their learning, 3.66 ±1.33 (with the median 4, 
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which corresponds to a very good mark). Students expressed that the least contribution in 
helping them learn was provided by discussion forums on the Type B system, and that this type of 
activity did not contribute much to better understand the learning topic, 2.26 ±1.30, with the 
median of 2 (corresponding to a sufficient mark). These findings show that students’ learning 
experiences are centered on problem-based learning and communication mainly with course 
instructors, and also with their peers. These results derive “nice to have functions” that should be 
considered within developing and modifying LOs. These “nice to have” functions point out that 
pedagogical methods that focus on increasing efficacy in communication with an instructor and 
classmates should be integrated with a Type B system. One possible approach to increase 
communication is to incorporate a social learning aspect within the Type B system, and foster 
problem-based learning through teamwork within this environment, guided and administered by 
the course instructor. 

Table 3 


Survey questions for evaluation of students’ degree of satisfaction 


Questions 

X± 

SD 

Med 

Pi_i How much do weekly course assignments help with the 
learning of course material 

3-66 ± 

1.13 

4 

Pi_2 What is your opinion about the quality of online learning 

3-31 ± 

1.2 

3 

Pi 3 What is the opinion of your surrounding about the quality of 
online learning (i.e., family, friends) 

3.11 ± 

1.21 

3 

Pi_4 How much have you established communication with your 
classmates 

3.40 ± 

1-37 

4 

Pi 5 How much have your interaction with classmates helped you out 

3.27 ± 

1-39 

3 

Pi_6 What is the degree of your current motivation to 
complete studies 

3.72 ± 

1.18 

4 

Pi_7 How much does the e-learning system help you in learning 

2.79 ± 

1.38 

3 

Pi_8 How much do discussion forums within the e-learning 
system help with your studying 

2.26 ± 

1-3 

2 

Pi_9 How much do interactive activities and exercises on e-learning 
system help you with studying 

2.63 ± 

1-37 

3 

Pi_io How important is your interaction and additional 
consults with instructors 

3 96 ± 

1.08 

4 


Table 4 

Result of survey questions based on the type of studying 
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Traditional n 

= 9i 


Online n = 93 




X± 

SD 

Med 

X± 

SD 

Med 

Pl_l 

3.67 ± 

1.04 

4 

3-65 ± 

1.21 

4 

Pi_2 

2.98 ± 

1.12 

3 

3-63 ± 

1.20*" 

4 

Pi _3 

2-73 ± 

1.08 

3 

3.48 ± 

1.23" 

4 

Pi_4 

4.00 ± 

I.06”* 

4 

2.81 ± 

1.38 

3 

Pi _5 

3-68 ± 

1.10*” 

4 

2.87 ± 

1-53 

3 

Pi_6 

3.64 ± 

1.18 

4 

3.80 ± 

1.18 

4 

Pi_7 

2-54 ± 

1-34 

2 

3-03 ± 

1 - 39 * 

3 

Pi_8 

1.89 ± 

l.ll 

1 

2.61 ± 

1.38" 

3 

Pi_9 

2.13 ± 

1.19 

2 

3.12 ± 

1 - 37 ’** 

3 

Pi_io 

3.76 ± 

1.11 

4 

4.16 ± 

1.02" 

5 


-p<0.001, ** -p<0.01, * -p<0.05 


Results from Table 4 were analyzed based on the variable “type of study” (online and traditional). 
It is clear that a statistically significant difference was established on most questions when two 
types of studies were examined. Traditional students were satisfied with established 
communication with classmates and they consider that this activity contributed the most in their 
learning process (Pi_4 and P15, pco.ooi). On the other hand, online students expressed the 
high importance of their interaction and additional consults with the instructor in their learning 
process (Pi_io, /ko.oi). At the same time, online students expressed the positive opinion about 
their activities on discussion forums, interactive activities, and exercises on the e-learning system 
(Pi_8 and Pi_9, p <0.001). Online students are also satisfied with the quality of online learning, 
and they think that their surrounding shares their opinion (Pi_2 and Pi_3, p< 0.001). These 
results are in line with previously drawn conclusions about adding “nice to have” functions that 
will emphasize pedagogical methods that foster communication and problem-based learning, 
along with teamwork and group activities. One can notice that traditional students expressed 
lower satisfaction with the e-learning system. Lower satisfaction can be attributed to the fact that 
traditional students used the e-learning system in blended-style learning with the main way of 
learning being class lecture and the e-learning as an additional source (not the only source). For 
establishing the statistical significance a Mann-Whitney test was used. 


Table 5 

Correlation matrix (correlation coefficients) of marks given on the survey questions (Pi) 



Pi_i 

Pi_ 2 

Pi_3 

Pi_4 

Pi_5 

Pi_6 

Pi_7 

Pi_8 

Pi_9 

Pl_10 
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Pl_l: How much do 
weekly course 
assignments help with 
the learning of course 
material 

1 

0.28 

*** 

0.28 

*** 

0.21 

** 

0.21 

** 

0.36 

*** 

0.33 

*** 

0.28 

*** 

0.32 

*** 

0.33 

*** 

Pl_2: What is your 
opinion about the quality 
of online learning 


1 

0.72 

*** 

0.02 

0.06 

0.35 

*** 

0.45 

*** 

0.46 

*** 

0.48 

*** 

0.18 

* 

Pl_3: What is the opinion 
of your surrounding 
about the quality of 
online learning (i.e. 
family, friends) 



1 

-0.01 

0.04 

0.30 

*** 

0.33 

*** 

0.35 

*** 

0.36 

*** 

0.12 

Pl_4: How much have 
you established 
communication with your 
classmates 





0.71 

*** 

0.12 

0.1 

0.08 

0 

0.08 

Pl_5: How much have 
your interactions with 
classmates helped you 
out 





■ 

0.13 

0.16 

* 

0.20 

** 

0.12 

0.16 

** 

Pt_6: What is the degree 
of your current 
motivation to complete 
studies 






1 

0.34 

*** 

0.28 

*** 

O.27 

*** 

0.22 

** 

Pi_7: How much does 
the e-learning system 
help you in learning 

■ 






1 

O.7O 

*** 

0.75 

*** 

0.19 

** 

Pl_8: How much do 
discussion forums within 
the e-learning system 
help with your studying 








1 

0.82 

*** 

0.19 

** 

Pl_9: How much do 
interactive activities and 
exercises on e-learning 
system help you with 
studying 









1 

0.30 

*** 

Pl_lO: How important is 
your interaction and 
additional consults with 
instructors 










1 


-p<0.001, ** -p<0.01, * -p<0.05 


It was of interest to analyze whether the correlation exists between the analyzed parameters in 
order to be able to better improve the system and predict future results of improvements. As 
shown in Table 5, it is evident that a statistically significant positive correlation exists between 
most questions. The degree of correlation was classified as follows: small (from 0.10 to 0.29), 
moderate (from 0.30 to 0.49), and high (from 0.50 to 1). These results show that students who 
are satisfied with discussion forums are also satisfied with interactive activities and exercises 
(Pi_8 and P19, p<o.ooi); students who have a positive attitude towards the type B system 
consider that discussion forums (Pi_7 and Pi_8, p<o.ooi), interactive activities, and exercises 
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(Pi_7 and P19, p<o.ooi) are very helpful for better success in studying and knowledge 
acquisition; students who have established good contacts with their classmates also feel that this 
connection contributes to their better success (Pi_4 and Pi_5, /xo.ooi); students who have a 
positive opinion about the quality of online learning also consider that their surrounding have the 
same opinion (Pi_2 and Pi_3, jko.ooi). 

Table 6 


Survey questions for evaluation of the degree of satisfaction about the quality of learning 
materials published on the Type B system 


Questions 

X± 

SD 

Med 

P2_1 Written lectures in PDF format 

4-51 ± 

0.96 

5 

P2_2 Lectures on e-learning system 

3.08 ± 

1.49 

3 

P2_3 Interactive questions on e-learning system 

2-59 ± 

1.44 

2 

P2_4 Discussion forum on e-learning system 

2.08 ± 

1-3 

2 

P2_5 Exercises 

4.09 ± 

1.12 

4 

P2_6 Additional literature 

3.42 ± 

1-35 

4 


A second part of the survey attempted to analyze how satisfied students were with published 
learning materials (Table 6). Based on the average marks from this part of the survey, it is 
noticeable that the highest mark was given for written lectures in PDF format, which actually 
represent summarized materials and leave students without utilizing the full functionalities of the 
Type B system, 4.51 ±0.96, with median of 5 (corresponding to the highest mark). The lowest 
mark was given to discussion forums, 2.08 ±1.30, with median of 2. This result shows that 
students did not take in the new system fully, and that their tendency is to study based on the 
learning methods from the Type A system. This result is significant, as it should lead to the 
improvements and further development of interactive and self-assessment activities on the Type 
B system, which would emphasize more learning outcomes and significance of each LO and its 
activities in the students’ learning process. On the other hand, even though the discussion forum 
received low marks, this does not necessarily mean that the forum represents a poor “methodical 
tool”, but it could also signify that not enough students have used the forum, so the discussion 
never developed between students, as well as between students and teachers. This also leads to 
the need for further development of “nice to have" characteristics of the LO system that should be 
implemented for further improvements by including more LAMS activities, and even be 
considered as “must have” in the next iteration. To step away from the old system, consideration 
should be given to remove residual functionalities of the Type A system because it detracts 
students from fully embracing the new system, such as written lectures in PDF format. 

Comparison of the results between Type A and Type B systems 
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In order to compare students’ satisfaction with the Type B system, the results were compared 
with the analysis previously conducted for the Type A system. Students’ level of satisfaction was 
classified as satisfied (when given marks 4 or 5) and unsatisfied (when given marks lower than 4). 
It can be noticed that students’ satisfaction decreased after transitioning to the Type B system. 
Moreover, these results point out that introducing a new system with significantly different 
functionalities, such as a Type B system, can in the short term decrease students’ satisfaction with 
learning experiences (Table 7). When introducing a new system, it can be expected that during 
the adjustment period, students may show the preferences towards the old style of learning; 
hence, it is important to carefully introduce novelties and tend towards their improvements. This 
is the reason for introducing “must have” functions as functions that are closer to the Type A 
system, to allow students an adjustment period. However, results show that this approach did not 
allow students to embrace new functionalities fully. Future steps in determining “nice to have” 
functions should move away from Type A functionalities, so that pedagogical methods that 
increase efficacy of learning through communication and problems solving can be further 
explored. 

Table 7 

Comparison of students’ satisfaction before and after the transition 



Satisfaction after the transition to Type 

B system 

Satisfaction based on the Type A system 

Online 

Traditional 

Online 

Traditional 

Unsatisfied 

Satisfied 

Unsatisfied 

Satisfied 

Unsatisfied 

Satisfied 

Unsatisfied 

Satisfied 

How much do 
weekly course 
assignments 
help with the 
learning of 
course material 

44.00 

56.00 

40.21 

59-79 

31.82 

68.18 

31-17 

68.83 

Have you 
managed to 
establish 
communication 
with your 
classmates 

66.00 

34-00 

26.80 

73-20 

60.61 

39-39 

14.29 

85-71 

How much 
have your 
interaction 
with classmates 
have helped 
you out 

62.00 

38.00 

38.14 

61.86 

59-09 

40.91 

22.08 

77.92 

Student’s 
opinion about 
online studies, 
in general 

40.00 

60.00 

71-13 

28.87 

16.67 

83-33 

50.65 

49-35 
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Surroundings 
opinion about 
the quality of 
online studies 

47.00 

53-oo 

77-32 

22.68 

31.82 

68.18 

59-09 

40.91 

Degree of 
current 
motivation to 
complete 
studies 

37-oo 

63.00 

37-H 

62.89 

22.73 

77.27 

20.78 

79.22 


Conclusions 

This paper proposed a hybrid transitional model for the implementation of an e-learning system 
that supports interactive learning through the sequencing of Los and interactive and self- 
assessment activities. The hybrid transitional model was based on the combination of waterfall 
and light agile methods. Content of learning materials from a non-LO oriented system served only 
as a baseline for developing content of LOs. The proposed hybrid transitional model required 
careful definition of the “must have” and “nice to have” functions. “Must have” functions have to 
be primarily based on the vital functionalities, without which the system cannot be put into 
operation. For the easier transition from the old to the new system, some of the “must have” 
functions were chosen based on the identified vital functions of the previous system. The main 
disadvantage of choosing functionalities of a non-LO oriented system to be a part of “must have” 
functions was instructors’ high level of familiarity with the previous system and the time and 
training needed for adoption of the new approach and structure of LOs. They in some way tried to 
maintain basic functions of the old system, which does not allow for full utilization of the new 
model. Furthermore, residuals from the new system also represented an obstacle among students, 
who were used to written lectures in PDF format and had a hard time accepting new structured, 
segmented, and reusable LOs. 

Development of LO content, peer review of LO, and multimedia development were recognized in 
the proposed model as a part of the Development/production ISO dimension. A suggested 
methodology for implementation of LO was analyzed using three components defined in the IEEE 
standard and through seven process categories of EN ISO/IEC 19796-1. This evaluation was also 
tightly connected with an assessment of an LO e-learning system from the students' point of view, 
and according to students’ survey analysis, improvements of LO content with “nice to have” 
functions are determined. In order to test the proposed model, the evaluation of the hybrid 
transitional model was conducted on the case study based on the students’ survey in order to 
verify “must have” and “nice to have” functions, the success of the transition, and to point out the 
key points of the system that can be improved. Analysis showed that students’ learning 
experiences were centered on problem-based learning (weekly and project assignments) and 
communication mainly with course instructors, and also their peers. Traditional students gave 
high importance to establishing good communication with their classmates, while online students 
gave the same importance to their communication with the instructor and additional consults 
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needed for their learning of the material. These findings point out that additional “nice to have” 
functions should be explored, in order to integrate pedagogical methods to suit the needs of the 
students. One approach to address this is to incorporate a social learning aspect within an LO- 
oriented system and foster problem-based learning and communication through teamwork and 
group activities, guided and administered by the course instructor. 

For the future iterations of improvements of the system, “must have“ requirements should focus 
on development and implementation of more appropriate self-assessment activities, as students 
find them important. These activities are directly linked to how students perceive the importance 
of the e-learning system in their studying. Furthermore, future work should explore satisfaction 
and inputs of the system evaluation from the faculty and institution management. Moreover, it 
will also be of great benefit to compare students’ academic success and results from non-LO and 
LO-oriented systems. These analyses will be of interest, once the institution has gone through 
several iteration of improvements and implementations of “nice to have” functions. 
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